Week 6

MACS 30405: Exploring Cultural Space
University of Chicago

Annoucement

Readings for week 7 will be released by the end of today.

Schedule update

Exercise 1 (PCA): released on Jan 13; due on Feb 9
Exercise 2 (FA/CA): released on Jan 27, due on Feb 9 (for FA) and/or 16 (for CA)
Exercise 3 (Text analysis): released on Feb 3, due on Feb 23
Exercise 4 (World embedding): to be released on Feb 10 and Feb 17, due on March 2
Exercise 5 (Anything2Vec): to be released on Feb 24, due on March 9

Possible final project

Identify, construct, and interpret cultural dimensions using multiple waves of a large-scale social survey. Try to answer the following questions:

Have the dimensional structures changed over time? Are there persistent dimensional structures over time? Or, do you observe any emergence or disappearance of dimensions?

Have any survey items changed their dimensional meanings over time? Which items change their dimensional loadings? Which becomes relevant or irrelevant? Does the change in their relevance change your understanding of the dimensions?

How have survey respondents changed? Do you observe any social demographic shift associated with any dimensional change? Could you explain any dimensional changes in terms of social demographic changes? Or, do you think more is going on in addition to social demographic change?

Finally, could you summarize the most significant shift that happened in the cultural space? Do you see only particular changes in terms of the dimensional loadings of some specific attitudes/tastes/behaviors whereas the dimensions mostly remain the same? Or, do you see some fundamental realignment of dimensions?

Example: In the United States, Liberals and conservatives used to be neutral in terms of confidence in science. However, survey results show that conservatives have dramatically lost confidence in science in recent decades. Social scientist have pondered two possible ways to explain the phenomenon:

How liberals and conservatives think about science has changed.
What defines liberalism vs. conservatism has changed.

Could you adjudicate which explanation is more accurate?

Reference: Kozlowski, A. C. (2022). How conservatives lost confidence in science: the role of ideological alignment in political polarization. Social Forces, 100(3), 1415-1443.

Possible data sources

General Social Survey
World Value Survey
Survey of Public Participation in the Arts (SPPA).

Saussure recap

Sign: the arbitrary relationship between signifiers and signified.

Language does not have concrete units

Language conditions thought: Thought without language is a “shapeless and indistinct mass.”
Language delimits units.

Signs as a sytem

Words do not have preexisting meaning. Language carves out meaning.
Signs are both arbitrary and differential: a sign is the counterpart of other signs.

Meaning of words

Words acquire value based on co-occurrence with other words that come before and after.
Words.that have something in common are associated.

Associative words

Word-embedding Models

Word co-occurrences can be thought of as a matrix.
You can try to apply PCA or CA to a word-document or word-author matrix.

World-embedding 1.0: Latent Sememtic Indexing (LSI)

Vector algebra can be then performed in world-embedding spaces

\(\text{Cosine Similarity}(w_1, w_2) = 1- \frac{\vec{w_1}\cdot\vec{w_2}}{||\vec{w_1}||||\vec{w_2}||}\)
\(\overrightarrow{king} + \overrightarrow{woman} - \overrightarrow{man} \approx \overrightarrow{queen}\)

Modern variant: Language models

Neural net

Word2Vec

Invented by Tomáš Mikolov’s team in 2013
The goal is not prediction but to learn high-quality a word embedding that has good geometric properties.
Fast, efficient and shallow.
Variants: CBOW and skip-gram

Skipgram

Task: find vector representation of words to maximize \[\begin{equation} \frac{1}{N} \sum _{i=1}^{N} \sum_{j=-k}^{k} \log p(w_{i+j}|w_{i}) \end{equation}\] where \(k\) is a window size.

\[\begin{equation} p(w_{i}|w_{j}) = \frac{\exp(\vec{u_{w_{i}}} \cdot \vec{v_{w_{j}}})}{\sum_{l=1}^{\mathcal{V}} \exp(\vec{u_{w_l}} \cdot \vec{v_{w_{j}}})}, \end{equation}\]

where \(v_{w_{j}}, u_{w_{i}} \in \mathbb{R}^d\) are the input and output vector representations of the word \(w_j\) and \(w_i\), and \(\mathcal{V}\) is the set of all unique vocabularies in the corpus.

The softmax function is essentially the same link function that is used in a multinomial logistic regression.
However, in practice, Mikolov’s skip-gram model never actually calculates the softmax function because computing it during training is very costly.

Apprximation

Word2vec implements 2 ways to speed up the apprximation: hierarchical softmax and skipgram and employs stochastic gradient descent (SGD) to find a solution.
Hierarchical softmax employs a Huffman Tree to reduce the computation for the softmax function.

Negative sampling (more popular)

Instead of modeling the conditional probability, negative sampling models

\[\begin{equation} p(y = 1|w_{i},w_{j}) = \text{sigmoid}(\vec{u_{w_{i}}}\cdot\vec{v_{w_{j}}}) \end{equation}\]

, where \(y\) is a binary variable indicating whether \(w_i\) and \(w_j\) co-occur in the training corpus.

The sigmoid function is just the link function of a logistic regression.

Evaluation

Because prediction is no longer the goal, embeddings learned from word2vec models are usually evaluated based on its performance in solving some analogy tests.